Many computer applications produce log files to record actions or events that occur as the applications execute. User events, function calls, system-level kernel calls, and file system calls are examples of application events that can be recorded in a log file.
Other computer applications may ingest these log files and use the log files to perform statistical analysis and generate metrics as output. For example, an application may produce logs of events from user interactions with websites which include advertisements. A second application or second process within the application may take these event logs and use them to produce metrics regarding advertising impressions such as a total number of impressions and unique cookie impressions in the given set of logs. Reports may be generated from these output metrics so it is important that the metrics are correct. However, it can be difficult to test the correctness of generated output metrics.
A conventional method for testing output may compare output from a new version of an application with output of a previous version of the application, given the same input. This testing method may catch regression bugs, but not existing bugs in the application. This method may also be impractical when a software developer has deliberately changed application behavior. Another testing method may hardcode input data and expected output. This method requires updating the tests when changes are made to the application, which is time consuming and can introduce bad data into tests. A different conventional testing method may require implementing features in test code and giving the same input to test and development implementations of the code and compare the output. This method may be useful when the production code is highly optimized, but there may not be a simple way to implement the same functionality. If there is no simple way to implement code functionality, this method may be costly to build and maintain, and hard to debug. An additional method may only check aspects of output that are easy to check, but this method may leave the most complicated part of an application's functionality untested.